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—  CamegieMclkia 

— Software  Engineering  Institute 

Introduction 

The  use  of  commercial  off-the-shelf  (COTS)  products  is  an 
increasingly  popular  approach  to  the  acquisition  of  major 
systems  throughout  the  government 

Results  are  mixed 

•  Some  succeed 

•  Some  don’t 

•  Others  have  a  lot  to  learn 
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—  CamegieMclkia 

— Software  Engineering  Institute 

Our  Comparison 

Selected  two  projects 
First-hand  experience  with  both 

Using  the  Software  Acquisition  Capability  Maturity  Model 
as  a  basis  for  comparison 
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GamegieMelloci 

Software  Engineering  Institute 

The  SA-CMM 


Level  2: 

Software  Acquisition  Planning 

Level  3: 

Solicitation 

Requirements  Development  and  Management 

Project  Management 

Contract  Tracking  and  Oversight 

Evaluation 

Transition  to  Support 

Process  Definition  and  Maintenance 

Level  4: 

User  Requirements 

Project  Performance  Management 

Contract  Performance  Management 

Acquisition  Risk  Management 

Training  Program  Management 

Quantitative  Process  Management 

Quantitative  Acquisition  Management 

Level  5: 

Continuous  Process  Improvement 

Acquisition  Innovation  Management 
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—  CamegieMclkia 

— Software  Engineering  Institute 

The  Projects 

Both: 

•  U.S.  Federal  agencies  that  fund  others 

•  Acquisition,  tailoring,  and  deployment  of  a  financial 
management  package 

•  Subject  to  political  pressures 

Project  A: 

•  Implementation  over  last  four  years 

•  Brought  vendor  on-board,  in  production 

•  Agency  operates  the  system 

Project  B: 

•  Implementation  over  last  year 

•  Engaged  system  integrator,  ready  for  pilot  testing  soon 

•  ASP  operates  the  system 
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GamegieMelloci 

Software  Engineering  Institute 

Software  Acquisition  Planning 


A:  B: 

•  Minimal  results  of  acquisition  • 
strategy/planning 

•  Reliance  on  GSA  contracts 

•  No  dedicated  acquisition 
organization  in-house 

-  no  in-house  documented 
procedures 

•  No  agency-wide  vision  for 
overall  automation  or  this  part 
of  it 


Planning  based  on  TSPR-like 
model 

Use  of  JFMIP  list 

No  dedicated  acquisition 
organization  in-house 

-  no  in-house  documented 
procedures 

High-level  buy-in  for  concept 
of  overall  automation 

-  externally  operated 

-  resistance  at  lower  levels 
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—  CiimpgipMt'llnn 

Software  Engineering  Institute 

Solicitation 

A:  B: 


•  Reliance  on  GSA  for  much  of  •  Performed  by  in-house 
this  expertise  program  office 

-  GSA  ran  the  solicitation 

-  very  positive  relationship 
and  results 
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—  CamegieMclkia 

— Software  Engineering  Institute 

Rqts  Development  and  Management 


A:  B: 

•  Agency  developed  a  very 
detailed  set  of  functional 
requirements 

-  based  on  another 
agency’s  successful 
solicitation  requirements 

-  liability  in  COTS 
acquisition 

•  Less  attention  to  non¬ 
functional  requirements, 
stakeholder  involvement,  and 
requirement  traceability 


Agency  developed  a  detailed 
set  of  functional  requirements 

-  developed  by  a  contractor 

-  needed  further  refinement 


Significant  attention  to  non¬ 
functional  requirements, 
stakeholder  involvement,  and 
requirement  traceability 
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GamegieMelloci 

Software  Engineering  Institute 

Project  Management 


A:  B: 

•  Very  weak  area 

-  no  team 

-  insufficient  resources 

-  leader  had  functional 
expertise,  not  software  or 
project  management 

•  Haphazard  attention  to 
issues  or  problems 

-purely  reactive 

•  Overall  lack  of  leadership 


Strong  program  management 

-  strong  PM  with  technical 
and  functional  expertise 

-  ability  to  choose  team 

-  resources  available  as 
needed 

Careful  planning  with  ability 
to  react  to  unforeseen 
circumstances 
Strong  leadership 
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GamegieMelloci 

Software  Engineering  Institute 

Contract  Tracking  &  Oversight 


A:  B: 

•  Three  confused  contracts: 

-  product  vendor 

-  infrastructure  integrator 

-  domain  consultant 

•  Often  follow,  not  lead  the 
contractors 

•  Incoherent  contract  change 
management 

•  No  one  in  agency 
experienced  in  contract 
management 

•  Few  plans  to  track  against 

•  No  systematic  recording  or 
tracking  of  problems 


Single  contractor 
-  experienced  integrator 
with  significant  experience 
in  the  product 

Considerable  direction  given 
to  contractor 
Close  management  of 
contractor 

PM  had  previous  acquisition 
experience 

Tasks  closely  tracked 
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—  CamegieMclkia 

Software  Engineering  Institute 

Evaluation 

A: 

•  No  evidence  of  any 
evaluation  requirements  or 
plan 

•  Unclear  how  they  decided 
acceptance 


B: 

•  Evaluation  requirements 
existed 

•  Contractor  was  best  match  to 
requirements 
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—  CamegieMclkia 

— Software  Engineering  Institute 

Transition  to  Support 


A: 

•  No  evidence  of  a  plan  for 
transition  or  support 


B: 

•  Integrating  contractor 
supports  the  system  for  the 
next  10  years 
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—  CamegieMclkia 

— Software  Engineering  Institute 

User  Requirements 

A:  B: 


•  Only  real  involvement  of  “end  •  Requirements  discussed  with 

users”  in  requirements  representatives  of  end  users 

determination:  the  guy  in 

charge  has  always  been  a 
functional 

•  No  organized  recording  of  •  User  requirements  managed 

user  requirements  using  requirements  tracking 

system 

•  No  organized  tracking  of  user 
requirements 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


page  13 


—  CamegieMclkia 

— Software  Engineering  Institute 

Project  Performance  Management 


A: 

•  No  process 

•  No  team  and  no  plan 

•  No  reviews 

•  No  risk  management 

•  No  project  management 


B: 

•  No  formal  process 

•  Strong  team  and  plan 

•  Weekly  reviews 

•  Risk  management  diffuse, 
but  strong 

•  Strong  project  management 
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—  CamegieMclkia 

— Software  Engineering  Institute 

Contract  Performance  Management 


A: 


B: 


•  Different  members  of 
different  parts  of  the  agency 
have  fairly  good  relations  with 
at  least  one  contractor 

•  No  evidence  of  contractor 
process  appraisals, 
evaluation  of  their 
performance,  or  proposals  for 
change 


Good  relationship  between 
agency  and  contractor  PMs 


Agency  organized  structure 
to  match  contractor 
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GamegieMelloci 

Software  Engineering  Institute 


Acquisition  Risk 

A: 

•  No  risk  management 

•  Not  even  any  backup  or 
contingency  plans  -  a 
necessity  for  COTS-based 
systems 


Management 

B: 

•  Many  different  sources  of  risk 
identification 

•  Strong  risk  mitigation  plans 


•  Program  relied  on  agency- 
based  risk  management  (plus 
PM’s  hot  list) 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


page  16 


—  CamegieMclkia 

— Software  Engineering  Institute 

Software  Acquisition  Planning 

A:  B: 


No  acquisition  management 
training 

-  have  been  content  to  let 
GSA  provide  all  expertise 


Experience  with  previous 
acquisitions 

-  intent  to  do  everything 
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—  CamegieMcIlao 

— Software  Engineering  Institute 

Practices  Not  Discussed 

Insufficient  information  to  compare  the  following  practice: 

•  Process  Definition  &  Maintenance 

The  following  practices  are  not  applicable: 

•  Quantitative  Process  Management 

•  Quantitative  Acquisition  Management 

•  Continuous  Process  Improvement 

•  Acquisition  Innovation  Management 
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—  CamegieMclkia 

— Software  Engineering  Institute 

Overall 

Agency  A  never  saw  itself  as  an  acquisition  organization 

•  No  acquisition  organization,  process,  or  plans 

•  No  vision 

•  No  project  management 

•  Grasped  at  COTS  products 

-  on  rebound  from  disastrous  custom  implementation 

Agency  B  also  not  an  acquisition  organization,  BUT 

•  Experienced  people 

•  Clear  vision 

•  Strong  project  management 

•  Careful  use  of  COTS  products 

-  filling  vacuums  in  enterprise  processes 

©  2003  by  Carnegie  Mellon  University  Version  1.0  page  19 


—  CamegieMclkia 

— Software  Engineering  Institute 

Reflections 

SA-CMM  has  provided  a  useful  vehicle  for  comparing  two 
acquisitions. 

Observation: 

SA-CMM  does  not  consider  the  future  operational  state. 
But  the  future  state  was  important  to  the  acquisition 
concept,  strategy,  and  planning  for  Project  B. 
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—  CamegieMclkia 

— Software  Engineering  Institute 

For  More  Information 


Tricia  Oberndorf 

412-268-6138 

po@sei.cmu.edu 


Pat  Place 

412-268-7746 

prp@sei.cmu.edu 
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